A Specification Coverage Metric Based on Dynamic Invariant Detection

Traditional coverage metrics measure the ability of test cases to fully exercise the source code. The assumption is that fully covered code will better test the correct behavior of the software. Unfortunately, there is no guarantee that the software will meet its specification. In this paper, we present a new coverage metric which measures the ability of the test cases to fully exercise the specification of the software. However, a specification is not required. Our approach allows the developer to iteratively develop and elaborate the test suite and the specification together. Using a dynamic invariant detector, we infer a specification from the execution of the software for the given test cases. If the test cases are inadequate, the specification will be inaccurate, and if the code is incorrect, the inferred specification will be incorrect. The tester can then refine the test suite or repair the software to improve the resulting specification. To evaluate this approach we applied it to three Java classes and their associated test suites. For each of the three classes, our method revealed incorrect and missing invariants in our specification, which implied limitations in our test suite. We were subsequently able to improve the test suite (and specification) by adding appropriate test cases. After applying our method several times, all incorrect invariants were removed, and our final test suite was more complete than the original.